genvid.toolbox.SDK

class genvid.toolbox.SDK(*args, **kwargs)

Bases: AWSTool, ConsulTemplateTool

Common commands for setting up the SDK.

This class wraps the ClusterAPI and BastionAPI for setting up a new Genvid SDK into the current cluster.

Changed in version 1.29.0: Moved load_config_template() to base class ConsulTemplateTool.

Changed in version 1.29.0: Moved merge_dict() to base class BaseTool.

Changed in version 1.30.0: Derived from ProjectTool instead of its ConsulTemplateTool ancestor.

Changed in version 1.30.0: Removed VaultTool and ConsulTool from the ancestry since they were already part of ConsulTemplateTool.

Changed in version 1.30.0: Moved load_nomad_job_template() to base class ProjectTool.

Changed in version 1.31.0: Moved update_config() to base class ProjectTool.

Changed in version 1.33.0: Loaded the binaries paths for a local cluster from GENVID_SDK_FOLDER/local-services/init folder, instead of hard coded. Also move the initialization from the static folders from setup_jobs() to setup().

Changed in version 1.33.0: Replace ProjectTool with ConfigurationLoader, now inheriting from ConsulTemplateTool instead.

Deprecated since version 1.44.0: Moved to new genvid.sdk package.

NAME = 'sdk'

The name of the logger for this tool. Must be redefined in children.

SDK_CONFIG_VERSION = '1.7.0'
cluster_id

The Cluster ID.

bastionAPI

Interface to the Bastion-API instance.

clusterAPI

Interface to the cluster Cluster-API instance.

cluster_vault_secret

Key in the bastion’s Vault where the cluster vault keys are saved.

GENVID_SDK_SETTINGS_SCHEMA

A mirror of GENVID_SDK_SETTINGS_SCHEMA containing the path to the default settings schema.

New in version 1.41.0.

property azure: AzureTool
property cloud_provider
get_remote_cluster_conn()

Return the cluster IP and private key.

get_remote_cluster_secrets()

Returns the cluster vault initialization secrets.

setup(bucket: Optional[str] = None, path: str = '/', *, region=None)

Starts the base SDK services.

Parameters:

  • bucket – The name of the bucket. Defaults to the return of get_s3_bucket_id().

  • path – The prefix key in the bucket for the images.

clean()

Stops the base SDK services, removes jobs and clean the config.

get_sdk_config() → dict

Get the default SDK configuration.

The configuration is found either in GENVID_SDK_FOLDER/local-services/config if the cluster_id is local, or GENVID_SDK_FOLDER/cloud-services/config otherwise. The config is loaded and parsed by consul-template first, and the jobs templates are expected to be found in a sibling folder named templates.

Changed in version 1.31.0: Load config from a folder.

get_config() → dict

Get the current cluster configuration.

set_config(config: Optional[dict] = None)

Merge the specified configuration with the current one. :param config: The configuration to merge.

clean_config()

Clean the configuration.

remove_config(config: dict)

Remove values from the configuration. :param config: The configuration to remove values from.

static update_dict_no_create(source, dest)
is_jobs_running()

Return True if a job is configured to be running.

start_jobs(jobs: Optional[List[str]] = None)

Start all jobs or the specified job. :param jobs: The list of jobs to start.

stop_jobs(jobs: Optional[List[str]] = None)

Stop all jobs or the specified job.

Parameters:

jobs – The list of jobs to stop.

property default_images_sdk_path

Get default S3 key prefix for SDK images.

New in version 1.20.0.

upload_images_sdk(bucket=None, path=None, update_config: bool = False, cloud_provider: Optional[str] = None, *, region=None)

Upload the images for the SDK.

Parameters:

  • bucket (str) – Bucket ID (default: get_s3_bucket_id()).

  • path – Key prefix to use in the bucket.

  • update_config (bool) – Update cluster images configuration.

  • region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Default path changes from ‘/’ to ‘/images/sdk-{version}/’.

Changed in version 1.20.0: New region parameter following the conventions in AWSTool.

Changed in version 1.26.0:

  • Added parameter region to target another region for the S3 bucket where the image will be uploaded.

  • If the bucket, named after the Bastion Id, does not exist, it will be created in the targeted region.

  • If it already exists but in a different region, the underlying code will raise an exception.

  • By default, the region used to create the bucket will be the region of the current AWS profile.

  • It can also be overriden by the AWS_DEFAULT_REGION environment variable.

  • If the region parameter is None and the bucket region already exists but its region differs from the one of the profile or AWS_DEFAULT_REGION environment variable, it will simply be used.

get_s3_bucket_id()

Get the S3 bucket ID for images.

Note

This method makes a global STS request to AWS. We disallow setting the region for this call to avoid confusion with the region of the bucket itself.

get_s3_bucket_with_bastion_id(bastion_id: str)

Get the S3 bucket ID for images using the current AWS account and the Bastion ID.

Note

This method makes a global STS request to AWS. We disallow setting the region for this call to avoid confusion with the region of the bucket itself.

update_images_config(bucket: Optional[str] = None, path: str = '/images/custom', *, delete: bool = False, region: Optional[str] = None)

Update the images configuration on the cluster.

Parameters:

  • bucket – Bucket ID (default: get_s3_bucket_id()).

  • path – Key prefix to use in the bucket.

  • delete – Do not update configuration, delete it instead.

  • region – Region containing the S3 bucket.

Warning

This method can only be called on a non-local cluster.

New in version 1.20.0.

upload_images(prefixes: List[str], bucket: Optional[str] = None, path: str = '/images/custom', update_config: bool = False, basedir: Optional[Union[Path, str]] = None, cloud_provider: Optional[str] = None, *, region: Optional[str] = None)

Upload the images with the specified prefixes.

Note

The image names must match DockerTool.RE_IMAGE_IMAGEID.

Parameters:

  • prefixes – Only upload the files with these prefixes.

  • bucket (str) – Bucket ID (default: get_s3_bucket_id()).

  • path – Key prefix to use in the bucket.

  • update_config (bool) – Update cluster images configuration.

  • basedir – Source directory (default: GENVID_SDK_FOLDER/images).

  • region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Change the default path from ‘/’ to ‘/images/custom’.

Changed in version 1.20.0:

  • Added the basedir parameter.

  • Augmented error handling.

  • New region parameter following the conventions in AWSTool.

  • Factored the update_config part of the method into its own method.

Changed in version 1.26.0:

  • Added parameter region to target another region for the S3 bucket where the image will be uploaded.

  • If the bucket, named after the Bastion Id, does not exist, it will be created in the targeted region.

  • If it already exists but in a different region, the underlying code will raise an exception.

  • By default, the region used to create the bucket will be the region of the current AWS profile.

  • It can also be overriden by the AWS_DEFAULT_REGION environment variable.

  • If the region parameter is None and the bucket region already exists but its region differs from the one of the profile or AWS_DEFAULT_REGION environment variable, it will simply be used.

create_nats_window_package()

Creates nats-window package if it does not exist.

Changed in version 1.32.0: Make it run on Linux using the binaries file.

get_images_config(bucket: Optional[str] = None, path: str = '/images/custom', *, region: Optional[str] = None) → dict
Retrieve images configurations from an S3 bucket or azure container

based on cloud provider.

Parameters:

  • bucket – Bucket ID .

  • path – Key prefix in the bucket (default: ‘/images/custom’).

  • region (str) – Region containing the S3 bucket/Azure Container.

Important

The image names must match DockerTool.RE_IMAGE_IMAGEID to be selected.

Returns:

Images configurations as a dict.

Changed in version 1.14.0: Changed the default path from ‘/’ to ‘/images/custom’.

Changed in version 1.20.0: New region parameter following the conventions in AWSTool.

clean_images(bucket: Optional[str] = None, path: str = '/images/custom', update_config: bool = False, *, region: Optional[str] = None)

Remove all images from an S3 bucket.

Parameters:

  • bucket – Bucket ID (default: get_s3_bucket_id()).

  • path – Key prefix to use in the bucket.

  • update_config (bool) – Update cluster images configuration.

  • region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Change the default path from ‘/’ to ‘/images/custom’.

Changed in version 1.20.0:

update_archive(archive: Union[Path, str], version: str = 'dev', destdir: Optional[Union[Path, str]] = None, clean: bool = True, keep: bool = False)

Update a binary archive to conform to the image format expected.

param archive:

The path to the archive file. The filename should always be in the form of name.archfmt.

param version:

Version of the archive.

param destdir:

Destination directory. Default to the same as the original archive.

param clean:

Clean old versions of the archive (the ones with the same name).

param keep:

Keep the original archives. You should almost always used it with a different destdir.

New in version 1.20.0.

Deprecated since version 1.30.0: Method moved to PackageArchiver

load_config(source: Union[Path, str], *, job_template_dir: Optional[Union[Path, str]], with_consul_template: bool, dryrun: bool = False) → dict

Loads the configuration file(s) (all files in a directory or a specific file) from the source.

Parameters:

  • source – Path to the configuration file/folder.

  • with_consul_template – Execute consul-template on the configuration file(s) (if source is a directory) before using it only if it is set to true.

  • job_template_dir – Path to Folder from where nomad templates should be loaded. If not mentioned, no jobs will be loaded.

  • dryrun – If True, only parse the source and returns the configuration.

Returns:

The configuration as a dictionary that can be used directly with set_config().

:raises FileNotFoundErrorIf the job_template_dir is not None and

is not a valid directory.

Changed in version 1.26.0: Fix support for pathlib.Path for arguments. Add load parameter

Changed in version 1.30.0: Refactored to use load_config_folder() instead of creating a new instance of ProjectTool.

Changed in version 1.33.0: Use the new render_template command instead of the local consul-template.

get_output_public_ips() → list

Retrieves all public ip’s from output of terraform module.

Returns:

List of all public IP Addressses

get_cloud_provider() → str

Retrieves the cloud provider from output of terraform module.

If no cloud provider is found in terraform’s cluster output, aws is returned as default.

Returns:

Cloud provider.

load_config_template(template: Union[Path, str], env: Optional[Mapping[str, str]] = None) → Any

Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

consul_template_once(*, use_consul: bool = False, use_vault: bool = False, env: Optional[Mapping[str, str]] = None, **kwargs) → ConsulTemplateOutput

Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

property consul_template

Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

class sdk.SDK

Implementation of genvid.toolbox.SDK